Tags: madeof:atoms, craft:sewing, FreeSoftWear
This post is a review for Computing Reviews for 10 things software developers should learn about learning , a article published in Communications of the ACMAs software developers, we understand the detailed workings of the different components of our computer systems. And probably due to how computers were presented since their appearance as digital brains in the 1940s we sometimes believe we can transpose that knowledge to how our biological brains work, be it as learners or as problem solvers. This article aims at making the reader understand several mechanisms related to how learning and problem solving actually work in our brains. It focuses on helping expert developers convey knowledge to new learners, as well as learners who need to get up to speed and start coding. The article s narrative revolves around software developers, but much of what it presents can be applied to different problem domains. The article takes this mission through ten points, with roughly the same space given to each of them, starting with wrong assumptions many people have about the similarities between computers and our brains. The first section, Human Memory Is Not Made of Bits, explains the brain processes of remembering as a way of strengthening the force of a memory ( reconsolidation ) and the role of activation in related network pathways. The second section, Human Memory Is Composed of One Limited and One Unlimited System, goes on to explain the organization of memories in the brain between long-term memory (functionally limitless, permanent storage) and working memory (storing little amounts of information used for solving a problem at hand). However, the focus soon shifts to how experience in knowledge leads to different ways of using the same concepts, the importance of going from abstract to concrete knowledge applications and back, and the role of skills repetition over time. Toward the end of the article, the focus shifts from the mechanical act of learning to expertise. Section 6, The Internet Has Not Made Learning Obsolete, emphasizes that problem solving is not just putting together the pieces of a puzzle; searching online for solutions to a problem does not activate the neural pathways that would get fired up otherwise. The final sections tackle the differences that expertise brings to play when teaching or training a newcomer: the same tools that help the beginner s productivity as training wheels will often hamper the expert user s as their knowledge has become automated. The article is written with a very informal and easy-to-read tone and vocabulary, and brings forward several issues that might seem like commonsense but do ring bells when it comes to my own experiences both as a software developer and as a teacher. The article closes by suggesting several books that further expand on the issues it brings forward. While I could not identify a single focus or thesis with which to characterize this article, the several points it makes will likely help readers better understand (and bring forward to consciousness) mental processes often taken for granted, and consider often-overlooked aspects when transmitting knowledge to newcomers.
This covers basically all my known omissions from last update except spellchecking of the Description field.
The X- style prefixes for field names are now understood and handled. This means the language server now considers XC-Package-Type the same as Package-Type.
More diagnostics:
- Fields without values now trigger an error marker
- Duplicated fields now trigger an error marker
- Fields used in the wrong paragraph now trigger an error marker
- Typos in field names or values now trigger a warning marker. For field names, X- style prefixes are stripped before typo detection is done.
- The value of the Section field is now validated against a dataset of known sections and trigger a warning marker if not known.
The "on-save trim end of line whitespace" now works. I had a logic bug in the server side code that made it submit "no change" edits to the editor.
The language server now provides "hover" documentation for field names. There is a small screenshot of this below. Sadly, emacs does not support markdown or, if it does, it does not announce the support for markdown. For now, all the documentation is always in markdown format and the language server will tag it as either markdown or plaintext depending on the announced support.
The language server now provides quick fixes for some of the more trivial problems such as deprecated fields or typos of fields and values.
Added more known fields including the XS-Autobuild field for non-free packages along with a link to the relevant devref section in its hover doc.
Despite its very limited feature set, I feel editing debian/control in emacs is now a much more pleasant experience. Coming back to the features that Otto requested, the above covers a grand total of zero. Sorry, Otto. It is not you, it is me.
- Diagnostics or linting of basic issues.
- Completion suggestions for all known field names that I could think of and values for some fields.
- Folding ranges (untested). This feature enables the editor to "fold" multiple lines. It is often used with multi-line comments and that is the feature currently supported.
- On save, trim trailing whitespace at the end of lines (untested). Might not be registered correctly on the server end.
Notable omission at this time:
- An error marker for syntax errors.
- An error marker for missing a mandatory field like Package or Architecture. This also includes Standards-Version, which is admittedly mandatory by policy rather than tooling falling part.
- An error marker for adding Multi-Arch: same to an Architecture: all package.
- Error marker for providing an unknown value to a field with a set of known values. As an example, writing foo in Multi-Arch would trigger this one.
- Warning marker for using deprecated fields such as DM-Upload-Allowed, or when setting a field to its default value for fields like Essential. The latter rule only applies to selected fields and notably Multi-Arch: no does not trigger a warning.
- Info level marker if a field like Priority duplicates the value of the Source paragraph.
- No errors are raised if a field does not have a value.
- No errors are raised if a field is duplicated inside a paragraph.
- No errors are used if a field is used in the wrong paragraph.
- No spellchecking of the Description field.
- No understanding that Foo and X[CBS]-Foo are related. As an example, XC-Package-Type is completely ignored despite being the old name for Package-Type.
- Quick fixes to solve these problems... :)
Obviously, the setup should get easier over time. The first three bullet points should eventually get resolved by merges and upload meaning you end up with an apt install command instead of them. For the editor part, I would obviously love it if we can add snippets for editors to make the automatically pick up the language server when the relevant file is installed.
- Build and install the deb of the main branch of pygls from https://salsa.debian.org/debian/pygls The package is in NEW and hopefully this step will soon just be a regular apt install.
- Build and install the deb of the rts-locatable branch of my python-debian fork from https://salsa.debian.org/nthykier/python-debian There is a draft MR of it as well on the main repo.
- Build and install the deb of the lsp-support branch of debputy from https://salsa.debian.org/debian/debputy
- Configure your editor to run debputy lsp debian/control as the language server for debian/control. This is depends on your editor. I figured out how to do it for emacs (see below). I also found a guide for neovim at https://neovim.io/doc/user/lsp. Note that debputy can be run from any directory here. The debian/control is a reference to the file format and not a concrete file in this case.
(with-eval-after-load 'eglot
(add-to-list 'eglot-server-programs
'(debian-control-mode . ("debputy" "lsp" "debian/control"))))
(with-eval-after-load "ox-md"
(org-export-define-derived-backend 'ik 'md
:translate-alist '((src-block . ik-src-block))
:menu-entry '(?m 1 ((?i "ikiwiki" ik-export-to-ikiwiki)))))
(defun ik-normalize-language (str)
(cond
((string-equal str "plait") "racket")
((string-equal str "smol") "racket")
(t str)))
(defun ik-src-block (src-block contents info)
"Transcode a SRC-BLOCK element from Org to beamer
CONTENTS is nil. INFO is a plist used as a communication
channel."
(let* ((body (org-element-property :value src-block))
(lang (ik-normalize-language (org-element-property :language src-block))))
(format "[[!format <span class="error">Error: unsupported page format %s</span>]]" lang body)))
(defun ik-export-to-ikiwiki
(&optional async subtreep visible-only body-only ext-plist)
"Export current buffer as an ikiwiki markdown file.
See org-md-export-to-markdown for full docs"
(require 'ox)
(interactive)
(let ((file (org-export-output-file-name ".mdwn" subtreep)))
(org-export-to-file 'ik file
async subtreep visible-only body-only ext-plist)))
queer.af
domain registration by the Taliban, the fragile and difficult nature of country-code top-level domains (ccTLDs) has once again been comprehensively demonstrated.
Since many people may not be aware of the risks, I thought I d give a solid explainer of the whole situation, and explain why you should, in general, not have anything to do with domains which are registered under ccTLDs.
https://
in your web browser s location bar).
It s the com in example.com
, or the af in queer.af
.
There are two kinds of TLDs: country-code TLDs (ccTLDs) and generic TLDs (gTLDs).
Despite all being TLDs, they re very different beasts under the hood.
queer.af
cancellation is interesting because, at the time the domain was reportedly registered, 2018, Afghanistan had what one might describe as, at least, a different political climate.
Since then, of course, things have changed, and the new bosses have decided to get a bit more active.
Those running queer.af
seem to have seen the writing on the wall, and were planning on moving to another, less fraught, domain, but hadn t completed that move when the Taliban came knocking.
.eu
, you have to be a resident of the EU.
When the UK ceased to be part of the EU, residents of the UK were no longer EU residents.
Cue much unhappiness, wailing, and gnashing of teeth when this was pointed out to Britons.
Some decided to give up their domains, and move to other parts of the Internet, while others managed to hold onto them by various legal sleight-of-hand (like having an EU company maintain the registration on their behalf).
In any event, all very unpleasant for everyone involved.
.sc
domain names from US$25 to US$75. No reason, no warning, just pay up .
.ly
.
These domain registrations weren t (and aren t) cheap, and it s hard to imagine that at least some of that money wasn t going to benefit the Gaddafi regime.
Similarly, the British Indian Ocean Territory, which has the io ccTLD, was created in a colonialist piece of chicanery that expelled thousands of native Chagossians from Diego Garcia.
Money from the registration of .io
domains doesn t go to the (former) residents of the Chagos islands, instead it gets paid to the UK government.
Again, I m not trying to suggest that all gTLD operators are wonderful people, but it s not particularly likely that the direct beneficiaries of the operation of a gTLD stole an island chain and evicted the residents.
.au
namespace some years ago.
Essentially, while a ccTLD may have geographic connotations now, there s not a lot of guarantee that they won t fall victim to scope creep in the future.
Finally, it might be somewhat safer to register under a ccTLD if you live in the location involved.
At least then you might have a better idea of whether your domain is likely to get pulled out from underneath you.
Unfortunately, as the .eu
example shows, living somewhere today is no guarantee you ll still be living there tomorrow, even if you don t move house.
In short, I d suggest sticking to gTLDs.
They re at least lower risk than ccTLDs.
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in RcppArmadillo version 0.12.8.0.0 (2024-02-06)
- Upgraded to Armadillo release 12.8.0 (Cortisol Injector)
- Faster detection of symmetric expressions by
pinv()
andrank()
- Expanded
shift()
to handle sparse matrices- Expanded
conv_to
for more flexible conversions between sparse and dense matrices- Added
cbrt()
- More compact representation of integers when saving matrices in CSV format
- Five non-user facing top-level include files have been removed (#432 closing #400 and building on #395 and #396)
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
In this talk Holger h01ger Levsen will give an overview about Reproducible Builds: How it started with a small BoF at DebConf13 (and before), then grew from being a Debian effort to something many projects work on together, until in 2021 it was mentioned in an Executive Order of the President of the United States. And of course, the talk will not end there, but rather outline where we are today and where we still need to be going, until Debian stable (and other distros!) will be 100% reproducible, verified by many. h01ger has been involved in reproducible builds since 2014 and so far has set up automated reproducibility testing for Debian, Fedora, Arch Linux, FreeBSD, NetBSD and coreboot.More information can be found on FOSDEM s own page for the talk, including a video recording and slides.
Our exploit path resulted in the ability to upload malicious PyTorch releases to GitHub, upload releases to [Amazon Web Services], potentially add code to the main repository branch, backdoor PyTorch dependencies the list goes on. In short, it was bad. Quite bad.The attack pivoted on PyTorch s use of self-hosted runners as well as submitting a pull request to address a trivial typo in the project s
README
file to gain access to repository secrets and API keys that could subsequently be used for malicious purposes.
archlinux-userland-fs-cmp
, the tool is supposed to be used from a rescue image (any Linux) with an Arch install mounted to, [for example], /mnt
. Crucially, however, at no point is any file from the mounted filesystem eval d or otherwise executed. Parsers are written in a memory safe language.
More information about the tool can be found on their announcement message, as well as on the tool s homepage. A GIF of the tool in action is also available.
SOURCE_DATE_EPOCH
code?
Chris Lamb started a thread on our mailing list summarising some potential problems with the source code snippet the Reproducible Builds project has been using to parse the SOURCE_DATE_EPOCH
environment variable:
I m not 100% sure who originally wrote this code, but it was probably sometime in the ~2015 era, and it must be in a huge number of codebases by now. Anyway, Alejandro Colomar was working on the shadow security tool and pinged me regarding some potential issues with the code. You can see this conversation here.Chris ends his message with a request that those with intimate or low-level knowledge of
time_t
, C types, overflows and the various parsing libraries in the C standard library (etc.) contribute with further info.
SOURCE_DATE_EPOCH
to document its interaction with distribution rebuilds. [ ].
254
and 255
to Debian but focusing on triaging and/or merging code from other contributors. This included adding support for comparing eXtensible ARchive (.XAR/.PKG) files courtesy of Seth Michael Larson [ ][ ], as well considerable work from Vekhir in order to fix compatibility between various and subtle incompatible versions of the progressbar libraries in Python [ ][ ][ ][ ]. Thanks!
arm64
architecture workers from 24 to 16. [ ]arm64
nodes when they hit an OOM (out of memory) state. [ ]real_year
variable to 2024 [ ] and bump various copyright years as well [ ].iptables
tool everywhere, else our custom rc.local
script fails. [ ]/srv/workspace/pbuilder
directory on boot. [ ]chroot-installation
jobs to a maximum of 4 concurrent runs. [ ][ ]armhf
architecture test infrastructure. This provided the incentive to replace the UPS batteries and consolidate infrastructure to reduce future UPS load. [ ]
Elsewhere in our infrastructure, however, Holger Levsen also adjusted the email configuration for @reproducible-builds.org
to deal with a new SMTP email attack. [ ]
cython
(nondeterminstic path issue)deluge
(issue with modification time of .egg
file)gap-ferret
, gap-semigroups
& gap-simpcomp
(nondeterministic config.log
file)grpc
(filesystem ordering issue )hub
(random)kubernetes1.22
& kubernetes1.23
(sort-related issue)kubernetes1.24
& kubernetes1.25
(go -trimpath
vs random issue)libjcat
(drop test files with random bytes)luajit
(Use new d
option for deterministic bytecode output)meson
[ ][ ] (sort the results from Python filesystem call)python-rjsmin
(drop GCC instrumentation artifacts)qt6-virtualkeyboard+others
(bug parallelism/race)SoapySDR
(parallelism-related issue)systemd
(sorting problem)warewulf
(CPIO modification time issue, etc.)guake
( Schroedinger file due to race condition)qhelpgenerator-qt5
(timezone localization; fix also merged upstream for QT6)sphinx
(search index doctitle
sorting)mm-common
package in Debian this was quickly fixed, however. [ ]
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
The clearance was entered initially with estimated import charges of 400.03, consisting of 387.83 VAT, and 12.20 disbursement fee. This original entry regrettably did not include the freight cost for calculating the VAT, and as such when submitted for final entry the VAT value was adjusted to include this and an amended invoice was issued for an additional 39.84. HMRC calculate the amount against which VAT is raised using the value of goods, insurance and freight, however they also may apply a VAT adjustment figure. The VAT Adjustment is based on many factors (Incidental costs in regards to a shipment), which includes charge for currency conversion if the invoice does not list values in Sterling, but the main is due to the inland freight from airport of destination to the final delivery point, as this charge varies, for example, from EMA to Edinburgh would be 150, from EMA to Derby would be 1, so each year UPS must supply HMRC with all values incurred for entry build up and they give an average which UPS have to use on the entry build up as the VAT Adjustment. The correct calculation for the import charges is therefore as follows: Goods value divided by exchange rate 2,489.53 EUR / 1.1683 = 2,130.89 GBP Duty: Goods value plus freight (%) 2,130.89 GBP + 5% = 2,237.43 GBP. That total times the duty rate. X 0 % = 0 GBP VAT: Goods value plus freight (100%) 2,130.89 GBP + 0 = 2,130.89 GBP That total plus duty and VAT adjustment 2,130.89 GBP + 0 GBP + 7.49 GBP = 2,348.08 GBP. That total times 20% VAT = 427.67 GBP As detailed above we must confirm that the final VAT charges applied to the shipment were correct, and that no refund of this is therefore due.This looks very like HMRC-originated nonsense. If only they had put it on the original bills! It s completely ridiculous that it took four months and near-litigation to obtain it. Disbursement fee One more thing. UPS billed me a 12 disbursement fee . When you import something, there s often tax to pay. The courier company pays that to the government, and the consignee pays it to the courier. Usually the courier demands it before final delivery, since otherwise they end up having to chase it as a debt. It is common for parcel companies to add a random fee of their own. As I note in my Particulars, there isn t any legal basis for this. In my own offer of settlement I proposed that UPS should:
State under what principle of English law (such as, what enactment or principle of Common Law), you levy the disbursement fee (or refund it).To my surprise they actually responded to this in their own settlement letter. (They didn t, for example, mention the harassment at all.) They said (emphasis mine):
A disbursement fee is a fee for amounts paid or processed on behalf of a client. It is an established category of charge used by legal firms, amongst other companies, for billing of various ancillary costs which may be incurred in completion of service. Disbursement fees are not covered by a specific law, nor are they legally prohibited. Regarding UPS disbursement fee this is an administrative charge levied for the use of UPS deferment account to prepay import charges for clearance through CDS. This charge would therefore be billed to the party that is responsible for the import charges, normally the consignee or receiver of the shipment in question. The disbursement fee as applied is legitimate, and as you have stated is a commonly used and recognised charge throughout the courier industry, and I can confirm that this was charged correctly in this instance.On UPS s analysis, they can just make up whatever fee they like. That is clearly not right (and I don t even need to refer to consumer protection law, which would also make it obviously unlawful). And, that everyone does it doesn t make it lawful. There are so many things that are ubiquitous but unlawful, especially nowadays when much of the legal system - especially consumer protection regulators - has been underfunded to beyond the point of collapse. Next time this comes up I might have a go at getting the fee back. (Obviously I ll have to pay it first, to get my parcel.) ParcelForce and Royal Mail I think this analysis doesn t apply to ParcelForce and (probably) Royal Mail. I looked into this in 2009, and I found that Parcelforce had been given the ability to write their own private laws: Schemes made under section 89 of the Postal Services Act 2000. This is obviously ridiculous but I think it was the law in 2009. I doubt the intervening governments have fixed it. Furniture Oh, yes, the actual furniture. The replacements arrived intact and are great :-).
pkgng
. It was
retired in favor of octavia around 2016.
Roughly 10 years later (2024-01-24), I found it in a drawer and, to my
surprised, it booted. After wrangling with a RS-232 USB adapter,
a null modem cable, and bit rates, I even logged in:
comBIOS ver. 1.33 20070103 Copyright (C) 2000-2007 Soekris Engineering.
net5501
0512 Mbyte Memory CPU Geode LX 500 Mhz
Pri Mas WDC WD800VE-00HDT0 LBA Xlt 1024-255-63 78 Gbyte
Slot Vend Dev ClassRev Cmd Stat CL LT HT Base1 Base2 Int
-------------------------------------------------------------------
0:01:2 1022 2082 10100000 0006 0220 08 00 00 A0000000 00000000 10
0:06:0 1106 3053 02000096 0117 0210 08 40 00 0000E101 A0004000 11
0:07:0 1106 3053 02000096 0117 0210 08 40 00 0000E201 A0004100 05
0:08:0 1106 3053 02000096 0117 0210 08 40 00 0000E301 A0004200 09
0:09:0 1106 3053 02000096 0117 0210 08 40 00 0000E401 A0004300 12
0:20:0 1022 2090 06010003 0009 02A0 08 40 80 00006001 00006101
0:20:2 1022 209A 01018001 0005 02A0 08 00 00 00000000 00000000
0:21:0 1022 2094 0C031002 0006 0230 08 00 80 A0005000 00000000 15
0:21:1 1022 2095 0C032002 0006 0230 08 00 00 A0006000 00000000 15
4 Seconds to automatic boot. Press Ctrl-P for entering Monitor.
______
____ __ ___ ___
Welcome to FreeBSD! __ '__/ _ \/ _ \
__ __/ __/
1. Boot FreeBSD [default] _ _ \___ \___
2. Boot FreeBSD with ACPI enabled ____ _____ _____
3. Boot FreeBSD in Safe Mode _ \ / ____ __ \
4. Boot FreeBSD in single user mode _) (___
5. Boot FreeBSD with verbose logging _ < \___ \
6. Escape to loader prompt _) ____) __
7. Reboot
____/ _____/ _____/
Select option, [Enter] for default
or [Space] to pause timer 5
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.4-RELEASE-p12 #5: Fri Jun 6 02:43:23 EDT 2014
root@roadkiller.anarc.at:/usr/obj/usr/src/sys/ROADKILL i386
gcc version 4.2.2 20070831 prerelease [FreeBSD]
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Geode(TM) Integrated Processor by AMD PCS (499.90-MHz 586-class CPU)
Origin = "AuthenticAMD" Id = 0x5a2 Family = 5 Model = a Stepping = 2
Features=0x88a93d<FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CLFLUSH,MMX>
AMD Features=0xc0400000<MMX+,3DNow!+,3DNow!>
real memory = 536870912 (512 MB)
avail memory = 506445824 (482 MB)
kbd1 at kbdmux0
K6-family MTRR support enabled (2 registers)
ACPI Error: A valid RSDP was not found (20101013/tbxfroot-309)
ACPI: Table initialisation failed: AE_NOT_FOUND
ACPI: Try disabling either ACPI or apic support.
cryptosoft0: <software crypto> on motherboard
pcib0 pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
Geode LX: Soekris net5501 comBIOS ver. 1.33 20070103 Copyright (C) 2000-2007
pci0: <encrypt/decrypt, entertainment crypto> at device 1.2 (no driver attached)
vr0: <VIA VT6105M Rhine III 10/100BaseTX> port 0xe100-0xe1ff mem 0xa0004000-0xa00040ff irq 11 at device 6.0 on pci0
vr0: Quirks: 0x2
vr0: Revision: 0x96
miibus0: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
vr0: Ethernet address: 00:00:24:cc:93:44
vr0: [ITHREAD]
vr1: <VIA VT6105M Rhine III 10/100BaseTX> port 0xe200-0xe2ff mem 0xa0004100-0xa00041ff irq 5 at device 7.0 on pci0
vr1: Quirks: 0x2
vr1: Revision: 0x96
miibus1: <MII bus> on vr1
ukphy1: <Generic IEEE 802.3u media interface> PHY 1 on miibus1
ukphy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
vr1: Ethernet address: 00:00:24:cc:93:45
vr1: [ITHREAD]
vr2: <VIA VT6105M Rhine III 10/100BaseTX> port 0xe300-0xe3ff mem 0xa0004200-0xa00042ff irq 9 at device 8.0 on pci0
vr2: Quirks: 0x2
vr2: Revision: 0x96
miibus2: <MII bus> on vr2
ukphy2: <Generic IEEE 802.3u media interface> PHY 1 on miibus2
ukphy2: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
vr2: Ethernet address: 00:00:24:cc:93:46
vr2: [ITHREAD]
vr3: <VIA VT6105M Rhine III 10/100BaseTX> port 0xe400-0xe4ff mem 0xa0004300-0xa00043ff irq 12 at device 9.0 on pci0
vr3: Quirks: 0x2
vr3: Revision: 0x96
miibus3: <MII bus> on vr3
ukphy3: <Generic IEEE 802.3u media interface> PHY 1 on miibus3
ukphy3: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
vr3: Ethernet address: 00:00:24:cc:93:47
vr3: [ITHREAD]
isab0: <PCI-ISA bridge> at device 20.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <AMD CS5536 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe000-0xe00f at device 20.2 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata0: [ITHREAD]
ata1: <ATA channel> at channel 1 on atapci0
ata1: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xa0005000-0xa0005fff irq 15 at device 21.0 on pci0
ohci0: [ITHREAD]
usbus0 on ohci0
ehci0: <AMD CS5536 (Geode) USB 2.0 controller> mem 0xa0006000-0xa0006fff irq 15 at device 21.1 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1 on ehci0
cpu0 on motherboard
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc8000-0xd27ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
ppc0: parallel port not found.
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
uart0: [FILTER]
uart0: console (19200,n,8,1)
uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 on isa0
uart1: [FILTER]
Timecounter "TSC" frequency 499903982 Hz quality 800
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ad0: 76319MB <WDC WD800VE-00HDT0 09.07D09> at ata0-master UDMA100
ugen0.1: <AMD> at usbus0
uhub0: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <AMD> at usbus1
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
GEOM: ad0s1: geometry does not match label (255h,63s != 16h,63s).
uhub0: 4 ports with 4 removable, self powered
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 4 ports with 4 removable, self powered
Trying to mount root from ufs:/dev/ad0s1a
The last log rotation is from 2016:
[root@roadkiller /var/log]# stat /var/log/wtmp
65 61783 -rw-r--r-- 1 root wheel 208219 1056 "Nov 1 05:00:01 2016" "Jan 18 22:29:16 2017" "Jan 18 22:29:16 2017" "Nov 1 05:00:01 2016" 16384 4 0 /var/log/wtmp
Interestingly, I switched between eicat and teksavvy on December
11th. Which year? Who knows!
Dec 11 16:38:40 roadkiller mpd: [eicatL0] LCP: authorization successful
Dec 11 16:41:15 roadkiller mpd: [teksavvyL0] LCP: authorization successful
Never realized those good old logs had a "oh dear forgot the year"
issue (that's something like Y2K except just "Y", I guess).
That was probably 2015, because the log dates from 2017, and the last
entry is from November of the year after the above:
[root@roadkiller /var/log]# stat mpd.log
65 47113 -rw-r--r-- 1 root wheel 193008 71939195 "Jan 18 22:39:18 2017" "Jan 18 22:39:59 2017" "Jan 18 22:39:59 2017" "Apr 2 10:41:37 2013" 16384 140640 0 mpd.log
It looks like the system was installed in 2010:
[root@roadkiller /var/log]# stat /
63 2 drwxr-xr-x 21 root wheel 2120 512 "Jan 18 22:34:43 2017" "Jan 18 22:28:12 2017" "Jan 18 22:28:12 2017" "Jul 18 22:25:00 2010" 16384 4 0 /
... so it lived for about 6 years, but still works after almost 14
years, which I find utterly amazing.
Another amazing thing is that there's tuptime installed on that
server! That is a software I thought I discovered later and then
sponsored in Debian, but turns out I was already using it then!
[root@roadkiller /var]# tuptime
System startups: 19 since 21:20:16 11/07/15
System shutdowns: 0 ok - 18 bad
System uptime: 85.93 % - 1 year, 11 days, 10 hours, 3 minutes and 36 seconds
System downtime: 14.07 % - 61 days, 15 hours, 22 minutes and 45 seconds
System life: 1 year, 73 days, 1 hour, 26 minutes and 20 seconds
Largest uptime: 122 days, 9 hours, 17 minutes and 6 seconds from 08:17:56 02/02/16
Shortest uptime: 5 minutes and 4 seconds from 21:55:00 01/18/17
Average uptime: 19 days, 19 hours, 28 minutes and 37 seconds
Largest downtime: 57 days, 1 hour, 9 minutes and 59 seconds from 20:45:01 11/22/16
Shortest downtime: -1 years, 364 days, 23 hours, 58 minutes and 12 seconds from 22:30:01 01/18/17
Average downtime: 3 days, 5 hours, 51 minutes and 43 seconds
Current uptime: 18 minutes and 23 seconds since 22:28:13 01/18/17
Actual up/down times:
[root@roadkiller /var]# tuptime -t
No. Startup Date Uptime Shutdown Date End Downtime
1 21:20:16 11/07/15 1 day, 0 hours, 40 minutes and 12 seconds 22:00:28 11/08/15 BAD 2 minutes and 37 seconds
2 22:03:05 11/08/15 1 day, 9 hours, 41 minutes and 57 seconds 07:45:02 11/10/15 BAD 3 minutes and 24 seconds
3 07:48:26 11/10/15 20 days, 2 hours, 41 minutes and 34 seconds 10:30:00 11/30/15 BAD 4 hours, 50 minutes and 21 seconds
4 15:20:21 11/30/15 19 minutes and 40 seconds 15:40:01 11/30/15 BAD 6 minutes and 5 seconds
5 15:46:06 11/30/15 53 minutes and 55 seconds 16:40:01 11/30/15 BAD 1 hour, 1 minute and 38 seconds
6 17:41:39 11/30/15 6 days, 16 hours, 3 minutes and 22 seconds 09:45:01 12/07/15 BAD 4 days, 6 hours, 53 minutes and 11 seconds
7 16:38:12 12/11/15 50 days, 17 hours, 56 minutes and 49 seconds 10:35:01 01/31/16 BAD 10 minutes and 52 seconds
8 10:45:53 01/31/16 1 day, 21 hours, 28 minutes and 16 seconds 08:14:09 02/02/16 BAD 3 minutes and 48 seconds
9 08:17:56 02/02/16 122 days, 9 hours, 17 minutes and 6 seconds 18:35:02 06/03/16 BAD 10 minutes and 16 seconds
10 18:45:18 06/03/16 29 days, 17 hours, 14 minutes and 43 seconds 12:00:01 07/03/16 BAD 12 minutes and 34 seconds
11 12:12:35 07/03/16 31 days, 17 hours, 17 minutes and 26 seconds 05:30:01 08/04/16 BAD 14 minutes and 25 seconds
12 05:44:26 08/04/16 15 days, 1 hour, 55 minutes and 35 seconds 07:40:01 08/19/16 BAD 6 minutes and 51 seconds
13 07:46:52 08/19/16 7 days, 5 hours, 23 minutes and 10 seconds 13:10:02 08/26/16 BAD 3 minutes and 45 seconds
14 13:13:47 08/26/16 27 days, 21 hours, 36 minutes and 14 seconds 10:50:01 09/23/16 BAD 2 minutes and 14 seconds
15 10:52:15 09/23/16 60 days, 10 hours, 52 minutes and 46 seconds 20:45:01 11/22/16 BAD 57 days, 1 hour, 9 minutes and 59 seconds
16 21:55:00 01/18/17 5 minutes and 4 seconds 22:00:04 01/18/17 BAD 11 minutes and 15 seconds
17 22:11:19 01/18/17 8 minutes and 42 seconds 22:20:01 01/18/17 BAD 1 minute and 20 seconds
18 22:21:21 01/18/17 8 minutes and 40 seconds 22:30:01 01/18/17 BAD -1 years, 364 days, 23 hours, 58 minutes and 12 seconds
19 22:28:13 01/18/17 20 minutes and 17 seconds
The last few entries are actually the tests I'm running now, it seems
this machine thinks we're now on 2017-01-18 at ~22:00, while we're
actually 2024-01-24 at ~12:00 local:
Wed Jan 18 23:05:38 EST 2017
FreeBSD/i386 (roadkiller.anarc.at) (ttyu0)
login: root
Password:
Jan 18 23:07:10 roadkiller login: ROOT LOGIN (root) ON ttyu0
Last login: Wed Jan 18 22:29:16 on ttyu0
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 8.4-RELEASE-p12 (ROADKILL) #5: Fri Jun 6 02:43:23 EDT 2014
Reminders:
* commit stuff in /etc
* reload firewall (in screen!):
pfctl -f /etc/pf.conf ; sleep 1
* vim + syn on makes pf.conf more readable
* monitoring the PPPoE uplink:
tail -f /var/log/mpd.log
Current problems:
* sometimes pf doesn't start properly on boot, if pppoe failed to come up, use
this to resume:
/etc/rc.d/pf start
it will kill your shell, but fix NAT (2012-08-10)
* babel fails to start on boot (2013-06-15):
babeld -D -g 33123 tap0 vr3
* DNS often fails, tried messing with unbound.conf (2014-10-05) and updating
named.root (2016-01-28) and performance tweaks (ee63689)
* asterisk and mpd4 are deprecated and should be uninstalled when we're sure
their replacements (voipms + ata and mpd5) are working (2015-01-13)
* if IPv6 fails, it's because netblocks are not being routed upstream. DHCPcd
should do this, but doesn't start properly, use this to resume (2015-12-21):
/usr/local/sbin/dhcpcd -6 --persistent --background --timeout 0 -C resolv.conf ng0
This machine is doomed to be replaced with the new omnia router, Indiegogo
campaign should ship in april 2016: http://igg.me/at/turris-omnia/x
(I really like the motd
I left myself there. In theory, I guess this
could just start connecting to the internet again if I still had the
same PPPoE/ADSL link I had almost a decade ago; obviously, I do
not.)
Not sure how the system figured the 2017 time: the onboard clock
itself believes we're in 1980, so clearly the CMOS battery has
(understandably) failed:
> ?
comBIOS Monitor Commands
boot [drive][:partition] INT19 Boot
reboot cold boot
download download a file using XMODEM/CRC
flashupdate update flash BIOS with downloaded file
time [HH:MM:SS] show or set time
date [YYYY/MM/DD] show or set date
d[b w d] [adr] dump memory bytes/words/dwords
e[b w d] adr value [...] enter bytes/words/dwords
i[b w d] port input from 8/16/32-bit port
o[b w d] port value output to 8/16/32-bit port
run adr execute code at adr
cmosread [adr] read CMOS RAM data
cmoswrite adr byte [...] write CMOS RAM data
cmoschecksum update CMOS RAM Checksum
set parameter=value set system parameter to value
show [parameter] show one or all system parameters
?/help show this help
> show
ConSpeed = 19200
ConLock = Enabled
ConMute = Disabled
BIOSentry = Enabled
PCIROMS = Enabled
PXEBoot = Enabled
FLASH = Primary
BootDelay = 5
FastBoot = Disabled
BootPartition = Disabled
BootDrive = 80 81 F0 FF
ShowPCI = Enabled
Reset = Hard
CpuSpeed = Default
> time
Current Date and Time is: 1980/01/01 00:56:47
Another bit of archeology: I had documented various outages with my
ISP... back in 2003!
[root@roadkiller ~/bin]# cat ppp_stats/downtimes.txt
11/03/2003 18:24:49 218
12/03/2003 09:10:49 118
12/03/2003 10:05:57 680
12/03/2003 10:14:50 106
12/03/2003 10:16:53 6
12/03/2003 10:35:28 146
12/03/2003 10:57:26 393
12/03/2003 11:16:35 5
12/03/2003 11:16:54 11
13/03/2003 06:15:57 18928
13/03/2003 09:43:36 9730
13/03/2003 10:47:10 23
13/03/2003 10:58:35 5
16/03/2003 01:32:36 338
16/03/2003 02:00:33 120
16/03/2003 11:14:31 14007
19/03/2003 00:56:27 11179
19/03/2003 00:56:43 5
19/03/2003 00:56:53 0
19/03/2003 00:56:55 1
19/03/2003 00:57:09 1
19/03/2003 00:57:10 1
19/03/2003 00:57:24 1
19/03/2003 00:57:25 1
19/03/2003 00:57:39 1
19/03/2003 00:57:40 1
19/03/2003 00:57:44 3
19/03/2003 00:57:53 0
19/03/2003 00:57:55 0
19/03/2003 00:58:08 0
19/03/2003 00:58:10 0
19/03/2003 00:58:23 0
19/03/2003 00:58:25 0
19/03/2003 00:58:39 1
19/03/2003 00:58:42 2
19/03/2003 00:58:58 5
19/03/2003 00:59:35 2
19/03/2003 00:59:47 3
19/03/2003 01:00:34 3
19/03/2003 01:00:39 0
19/03/2003 01:00:54 0
19/03/2003 01:01:11 2
19/03/2003 01:01:25 1
19/03/2003 01:01:48 1
19/03/2003 01:02:03 1
19/03/2003 01:02:10 2
19/03/2003 01:02:20 3
19/03/2003 01:02:44 3
19/03/2003 01:03:45 3
19/03/2003 01:04:39 2
19/03/2003 01:05:40 2
19/03/2003 01:06:35 2
19/03/2003 01:07:36 2
19/03/2003 01:08:31 2
19/03/2003 01:08:38 2
19/03/2003 01:10:07 3
19/03/2003 01:11:05 2
19/03/2003 01:12:03 3
19/03/2003 01:13:01 3
19/03/2003 01:13:58 2
19/03/2003 01:14:59 5
19/03/2003 01:15:54 2
19/03/2003 01:16:55 2
19/03/2003 01:17:50 2
19/03/2003 01:18:51 3
19/03/2003 01:19:46 2
19/03/2003 01:20:46 2
19/03/2003 01:21:42 3
19/03/2003 01:22:42 3
19/03/2003 01:23:37 2
19/03/2003 01:24:38 3
19/03/2003 01:25:33 2
19/03/2003 01:26:33 2
19/03/2003 01:27:30 3
19/03/2003 01:28:55 2
19/03/2003 01:29:56 2
19/03/2003 01:30:50 2
19/03/2003 01:31:42 3
19/03/2003 01:32:36 3
19/03/2003 01:33:27 2
19/03/2003 01:34:21 2
19/03/2003 01:35:22 2
19/03/2003 01:36:17 3
19/03/2003 01:37:18 2
19/03/2003 01:38:13 3
19/03/2003 01:39:39 2
19/03/2003 01:40:39 2
19/03/2003 01:41:35 3
19/03/2003 01:42:35 3
19/03/2003 01:43:31 3
19/03/2003 01:44:31 3
19/03/2003 01:45:53 3
19/03/2003 01:46:48 3
19/03/2003 01:47:48 2
19/03/2003 01:48:44 3
19/03/2003 01:49:44 2
19/03/2003 01:50:40 3
19/03/2003 01:51:39 1
19/03/2003 11:04:33 19
19/03/2003 18:39:36 2833
19/03/2003 18:54:05 825
19/03/2003 19:04:00 454
19/03/2003 19:08:11 210
19/03/2003 19:41:44 272
19/03/2003 21:18:41 208
24/03/2003 04:51:16 6
27/03/2003 04:51:20 5
30/03/2003 04:51:25 5
31/03/2003 08:30:31 255
03/04/2003 08:30:36 5
06/04/2003 01:16:00 621
06/04/2003 22:18:08 17
06/04/2003 22:32:44 13
09/04/2003 22:33:12 28
12/04/2003 22:33:17 6
15/04/2003 22:33:22 5
17/04/2003 15:03:43 18
20/04/2003 15:03:48 5
23/04/2003 15:04:04 16
23/04/2003 21:08:30 339
23/04/2003 21:18:08 13
23/04/2003 23:34:20 253
26/04/2003 23:34:45 25
29/04/2003 23:34:49 5
02/05/2003 13:10:01 185
05/05/2003 13:10:06 5
08/05/2003 13:10:11 5
09/05/2003 14:00:36 63928
09/05/2003 16:58:52 2
11/05/2003 23:08:48 2
14/05/2003 23:08:53 6
17/05/2003 23:08:58 5
20/05/2003 23:09:03 5
23/05/2003 23:09:08 5
26/05/2003 23:09:14 5
29/05/2003 23:00:10 3
29/05/2003 23:03:01 10
01/06/2003 23:03:05 4
04/06/2003 23:03:10 5
07/06/2003 23:03:38 28
10/06/2003 23:03:50 12
13/06/2003 23:03:55 6
14/06/2003 07:42:20 3
14/06/2003 14:37:08 3
15/06/2003 20:08:34 3
18/06/2003 20:08:39 6
21/06/2003 20:08:45 6
22/06/2003 03:05:19 138
22/06/2003 04:06:28 3
25/06/2003 04:06:58 31
28/06/2003 04:07:02 4
01/07/2003 04:07:06 4
04/07/2003 04:07:11 5
07/07/2003 04:07:16 5
12/07/2003 04:55:20 6
12/07/2003 19:09:51 1158
12/07/2003 22:14:49 8025
15/07/2003 22:14:54 6
16/07/2003 05:43:06 18
19/07/2003 05:43:12 6
22/07/2003 05:43:17 5
23/07/2003 18:18:55 183
23/07/2003 18:19:55 9
23/07/2003 18:29:15 158
23/07/2003 19:48:44 4604
23/07/2003 20:16:27 3
23/07/2003 20:37:29 1079
23/07/2003 20:43:12 342
23/07/2003 22:25:51 6158
Fascinating.
I suspect the (IDE!) hard drive might be failing as I saw two new
files created in /var
that I didn't remember seeing before:
-rw-r--r-- 1 root wheel 0 Jan 18 22:55 3@T3
-rw-r--r-- 1 root wheel 0 Jan 18 22:55 DY5
So I shutdown the machine, possibly for the last time:
Waiting (max 60 seconds) for system process bufdaemon' to stop...done
Waiting (max 60 seconds) for system process syncer' to stop...
Syncing disks, vnodes remaining...3 3 0 1 1 0 0 done
All buffers synced.
Uptime: 36m43s
usbus0: Controller shutdown
uhub0: at usbus0, port 1, addr 1 (disconnected)
usbus0: Controller shutdown complete
usbus1: Controller shutdown
uhub1: at usbus1, port 1, addr 1 (disconnected)
usbus1: Controller shutdown complete
The operating system has halted.
Please press any key to reboot.
I'll finally note this was the last FreeBSD server I personally
operated. I also used FreeBSD to setup the core routers at
Koumbit but those were replaced with Debian recently as
well.
Thanks Soekris, that was some sturdy hardware. Hopefully this new
Protectli router will live up to that "decade
plus" challenge.
Not sure what the fate of this device will be: I'll bring it to the
next Montreal Debian & Stuff to see if anyone's interested,
contact me if you can't show up and want this thing.
/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G4 /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Organization Validation Secure Server CA /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 /C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./OU=http://certs.starfieldtech.com/repository//CN=Starfield Secure Certificate Authority - G2 /C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020Rather than try to work with raw issuers (because, as Andrew Ayer says, The SSL Certificate Issuer Field is a Lie), I mapped these issuers to the organisations that manage them, and summed the counts for those grouped issuers together.
Issuer | Compromised Count |
---|---|
Sectigo | 170 |
ISRG (Let's Encrypt) | 161 |
GoDaddy | 141 |
DigiCert | 81 |
GlobalSign | 46 |
Entrust | 3 |
SSL.com | 1 |
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
Entrust | 23,166 | 3 | 1 in 7,722 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
SELECT SUM(sub.NUM_ISSUED[2] - sub.NUM_EXPIRED[2]) FROM ( SELECT ca.name, max(coalesce(coalesce(nullif(trim(cc.SUBORDINATE_CA_OWNER), ''), nullif(trim(cc.CA_OWNER), '')), cc.INCLUDED_CERTIFICATE_OWNER)) as OWNER, ca.NUM_ISSUED, ca.NUM_EXPIRED FROM ccadb_certificate cc, ca_certificate cac, ca WHERE cc.CERTIFICATE_ID = cac.CERTIFICATE_ID AND cac.CA_ID = ca.ID GROUP BY ca.ID ) sub WHERE sub.name ILIKE '%Amazon%' OR sub.name ILIKE '%CloudFlare%' AND sub.owner = 'DigiCert';The number I get from running that query is 104,316,112, which should be subtracted from DigiCert s total issuance figures to get a more accurate view of what DigiCert s regular customers do with their private keys. When I do this, the compromise rates table, sorted by the compromise rate, looks like this:
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
"Regular" DigiCert | 40,397,363 | 81 | 1 in 498,732 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
All DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
The less humans have to do with certificate issuance, the less likely they are to compromise that certificate by exposing the private key. While it may not be surprising, it is nice to have some empirical evidence to back up the common wisdom. Fully-managed TLS providers, such as CloudFlare, AWS Certificate Manager, and whatever Azure s thing is called, is the platonic ideal of this principle: never give humans any opportunity to expose a private key. I m not saying you should use one of these providers, but the security approach they have adopted appears to be the optimal one, and should be emulated universally. The ACME protocol is the next best, in that there are a variety of standardised tools widely available that allow humans to take themselves out of the loop, but it s still possible for humans to handle (and mistakenly expose) key material if they try hard enough. Legacy issuance methods, which either cannot be automated, or require custom, per-provider automation to be developed, appear to be at least four times less helpful to the goal of avoiding compromise of the private key associated with a certificate.
So these "simple" files have way too many combinations of how they can be interpreted. I figured it would be helpful if debputy could highlight these difference, so I added support for those as well. Accordingly, debian/install is tagged with multiple tags including dh-executable-config and dh-glob-after-execute. Then, I added a datatable of these tags, so it would be easy for people to look up what they meant. Ok, this seems like a closed deal, right...?
- Will the debhelper use filearray, filedoublearray or none of them to read the file? This topic has about 2 bits of entropy.
- Will the config file be executed if it is marked executable assuming you are using the right compat level? If it is executable, does dh-exec allow renaming for this file? This topic adds 1 or 2 bit of entropy depending on the context.
- Will the config file be subject to glob expansions? This topic sounds like a boolean but is a complicated mess. The globs can be handled either by debhelper as it parses the file for you. In this case, the globs are applied to every token. However, this is not what dh_install does. Here the last token on each line is supposed to be a directory and therefore not subject to globs. Therefore, dh_install does the globbing itself afterwards but only on part of the tokens. So that is about 2 bits of entropy more. Actually, it gets worse...
- If the file is executed, debhelper will refuse to expand globs in the output of the command, which was a deliberate design choice by the original debhelper maintainer took when he introduced the feature in debhelper/8.9.12. Except, dh_install feature interacts with the design choice and does enable glob expansion in the tool output, because it does so manually after its filedoublearray call.
You can help yourself and others to better results by using the declarative way rather than using debian/rules, which is the bane of all introspection!
- When determining which commands are relevant, using Build-Depends: dh-sequence-foo is much more reliable than configuring it via the Turing complete configuration we call debian/rules.
- When debhelper commands use NOOP promise hints, dh_assistant can "see" the config files listed those hints, meaning the file will at least be detected. For new introspectable hint and the debputy plugin, it is probably better to wait until the dust settles a bit before adding any of those.
pyproject.toml
files, I wanted to investigate how the popularity of build
backends used in pyproject.toml
files evolved over the years since the
introduction of PEP-0517 in 2015.
Getting the data
Tom Forbes provides a huge
dataset that contains information
about every file within every release uploaded to PyPI. To
get the current dataset, we can use:
curl -L --remote-name-all $(curl -L "https://github.com/pypi-data/data/raw/main/links/dataset.txt")
describe select * from '*.parquet';
column_name column_type null
varchar varchar varchar
project_name VARCHAR YES
project_version VARCHAR YES
project_release VARCHAR YES
uploaded_on TIMESTAMP YES
path VARCHAR YES
archive_path VARCHAR YES
size UBIGINT YES
hash BLOB YES
skip_reason VARCHAR YES
lines UBIGINT YES
repository UINTEGER YES
11 rows 6 columns
pyproject.toml
files that are in the project s root directory. Since we ll still have to
download the actual files, we need to get the path
and the repository
to
construct the corresponding URL to the mirror that contains all files in a
bunch of huge git repositories. Some files are not available on the mirrors; to
skip these, we only take files where the skip_reason
is empty. We also care
about the timestamp of the upload (uploaded_on
) and the hash
to avoid
processing identical files twice:
select
path,
hash,
uploaded_on,
repository
from '*.parquet'
where
skip_reason == '' and
lower(string_split(path, '/')[-1]) == 'pyproject.toml' and
len(string_split(path, '/')) == 5
order by uploaded_on desc
repository
and path
, we can now construct an URL from which we
can fetch the actual file for further processing:
url = f"https://raw.githubusercontent.com/pypi-data/pypi-mirror- repository /code/ path "
pyproject.toml
files and parse them to read
the build-backend
into a dictionary mapping the file-hash
to the build
backend. Downloads on GitHub are rate-limited, so downloading 1.2M files
will take a couple of days. By skipping files with a hash we ve already
processed, we can avoid downloading the same file more than once, cutting the
required downloads by circa 50%.
Results
Assuming the data is complete and my analysis is sound, these are the findings:
There is a surprising amount of build backends in use, but the overall amount
of uploads per build backend decreases quickly, with a long tail of single
uploads:
>>> results.backend.value_counts()
backend
setuptools 701550
poetry 380830
hatchling 56917
flit 36223
pdm 11437
maturin 9796
jupyter 1707
mesonpy 625
scikit 556
...
postry 1
tree 1
setuptoos 1
neuron 1
avalon 1
maturimaturinn 1
jsonpath 1
ha 1
pyo3 1
Name: count, Length: 73, dtype: int64
pyproject.toml
files. During that early
period, Flit started as the most popular build backend, but was eventually
displaced by Setuptools and Poetry.
Between 2020 and 2020, the overall usage of pyproject.toml
files increased
significantly. By the end of 2022, the share of Setuptools peaked at 70%.
After 2020, other build backends experienced a gradual rise in popularity.
Amongh these, Hatch emerged as a notable contender, steadily gaining
traction and ultimately stabilizing at 10%.
We can also look into the absolute distribution of build backends over time:
The plot shows that Setuptools has the strongest growth trajectory, surpassing
all other build backends. Poetry and Hatch are growing at a comparable rate,
but since Hatch started roughly 4 years after Poetry, it s lagging behind in
popularity. Despite not being among the most widely used backends anymore, Flit
maintains a steady and consistent growth pattern, indicating its enduring
relevance in the Python packaging landscape.
The script for downloading and analyzing the data can be found in my GitHub
repository. It contains the results of the duckb query (so you
don t have to download the full dataset) and the pickled dictionary, mapping
the file hashes to the build backends, saving you days for downloading and
analyzing the pyproject.toml
files yourself.
Next.